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Abstract 


A  central  challenge  in  defense  acquisition  is  the  development  of  accurate  cost  and 
schedule  estimates.  The  lack  of  discipline  in  estimating  and  unrealistic  expectations 
in  the  early  phases  of  programs  have  been  often  cited  as  common  causes  for  poor 
performance  of  large  programs  (GAO,  2004,  2006).  Initial  estimates  provided  by 
contractors  are  known  to  “anchor”  expectations  (Aranda  &  Easterbrook,  2005),  even 
when  changes  in  personnel,  technology,  or  budgetary  priorities  can  affect  the 
performance  of  a  program.  We  examine  the  use  of  prediction  markets  as  a  tool  for 
generating  schedule  estimates  as  a  supplement  to  existing  estimation 
methodologies. 
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Report  Summary 

A  central  challenge  in  defense  acquisition  is  the  development  of  accurate  cost  and 
schedule  estimates.  The  lack  of  discipline  in  estimating  and  unrealistic  expectations  in  the 
early  phases  of  programs  have  been  often  cited  as  common  causes  for  poor  performance  of 
large  programs  (GAO,  2004,  2006).  Initial  estimates  provided  by  contractors  are  known  to 
“anchor”  expectations  (Aranda  &  Easterbrook,  2005),  even  when  changes  in  personnel, 
technology,  or  budgetary  priorities  can  affect  the  performance  of  a  program.  We  examine 
the  use  of  prediction  markets  as  a  tool  for  generating  schedule  estimates  as  a  supplement 
to  existing  estimation  methodologies.  A  prediction  market  provides  an  environment  for 
traders  to  buy  and  sell  contracts  whose  value  is  tied  to  an  uncertain  future  event,  such  as 
the  duration  of  a  weapons  system  acquisition.  Most  notably  used  today  for  predicting 
election  outcomes,  prediction  markets  are  used  to  forecast  product  sales,  movie  box  office 
returns,  terrorist  attacks,  and  sporting  events  (Wolfers  &  Zitzewitz,  2004). 

A  prediction  market  is  a  means  of  forecasting  some  unknown  future  condition  of  the 
world.  In  a  prediction  market,  buyers  and  sellers  trade  contracts  and  money  for  contracts 
whose  payoff  depends  on  the  future  state  (Wolfers  &  Zitzewitz,  2004).  If  the  market  is  well 
functioning,  contract  prices  reflect  the  collective  wisdom  of  the  market  participants.  There 
are  three  primary  types  of  prediction  markets.1  Much  of  the  enthusiasm  for  prediction 
markets  derives  from  the  efficient  markets  hypothesis.  In  a  truly  efficient  prediction  market, 
the  market  price  of  a  prediction  market  contract  will  best  summarize  traders’  beliefs  about 
the  probability  of  the  event’s  occurrence.  Efficient  prediction  markets  should  outperform 
available  polls  and  other  forecasting  mechanisms. 

We  anticipate  prediction  markets  to  outperform  existing  defense  acquisition 
estimation  techniques  (i.e.,  parametric,  analogy,  activity-based)  for  cases  in  which  there  is 
ample  “soft,”  relative  to  “hard,”  information,  and  information  is  broadly  and  unevenly  held  by 
diverse  actors.  Examples  of  such  circumstances  include  one-of-a-kind  acquisitions  in  which 
limited  historical  information  exists,  and  acquisitions  that  are  prone  to  performance  impacts 
to  external  events.  Modifications  to  the  design,  shifts  in  program  personnel,  or  changes  in 
the  political  landscape  may  have  significant  impacts  on  the  cost  and  duration  of  a  program. 
Existing  cost-estimation  techniques  are  not  sensitive  to  these  types  of  changes  because  (1 ) 
most  Cost  Estimating  Relationships  (CERs)  are  based  on  technical  factors,  rather  than 
programmatic  “soft”  factors;  (2)  cost  estimates  are  not  dynamically  updated  as  a  program 
evolves,  making  the  original  estimate  outdated  as  soon  as  the  climate  changes;  and  (3)  cost 
estimates  are  a  manifestation  of  a  few  decision  makers,  often  under  tremendous  time 
pressure,  working  with  limited  and,  perhaps  biased,  information. 

By  shifting  the  paradigm  from  estimating  by  individuals  to  estimating  by  groups,  we 
can  harness  the  wisdom  of  crowds  by  capturing  their  collective  intelligence.  A  prediction 
market  facilitates  the  aggregation  of  data  from  diverse  and  independent  sources,  yielding 
more  accurate  forecasts.  The  prediction  markets’  value  is  grounded  in  several  factors. 

First,  they  provide  a  way  to  leverage  the  wisdom  of  crowds  by  aggregating  information  from 
diverse  sources.  Studies  have  shown  that  under  the  right  circumstances,  prediction  markets 
are  quite  accurate,  and  often  more  so  than  even  the  most  accurate  individual  forecasters 
(Surowiecki,  2005;  Griffiths  &  Tenenbaum,  2006).  Second,  they  mitigate  decision  biases 
stemming  from  pressures  to  “price  to  win”  and  hide  information.  Third,  they  enable  frequent 
sampling  of  information,  which  makes  them  more  responsive  to  environmental  changes. 
Finally,  prediction  markets  provide  incentives  for  traders  to  seek  out  additional  information. 


1  The  following  discussion  of  prediction  market  types  comes  from  Wolfers  and  Zitzewitz  (2004). 
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As  with  any  markets,  prediction  markets  may  fail — and  produce  inaccurate 
forecasts — if  not  properly  designed  and  executed.  Like  markets  generally,  effective 
prediction  markets  require  a  sufficient  number  of  buyers  and  sellers,  well  informed  about 
each  other  and  their  resources,  and  a  mechanism  through  which  they  can  exchange 
resources  under  fully  specified,  clear,  and  enforceable  contracts.  We  briefly  discuss  some 
potential  challenges  in  prediction  market  design  before  discussing  more  specific  prediction 
market  design  principles  and  the  conditions  for  making  them  successful. 
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Cost  Commitment  on  Projects 
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Blanchard,  B.,  Fabrycky,  W.,  Systems  Engineering  &  Analysis,  Prentice  Hall,  1998. 


Cone  of  Uncertainty 
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Source:  Barry  Boehm  (USC) 


How  Cost  Modeling  is  Done  Today 


I KSLOC 
1  Weight 
Requirements 


Size 

Drivers  ■ 

Effort  H 
Multipliers 


i  Product  complexity 
■  Personnel  Experience 
1  Architecture  understanding 
Legacy  integration  issues 


Cost  Model 


Calibration 


Effort 
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Prediction  Markets 


Technical  factors  rather 
than  “soft"  factors 


- \ 

Not  dynamically 

updated  as  the  program 
evolves 


Few  decision  makers 

under  time  pressure  or 

biased 
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Value  Proposition 


I  nformation  leveraged  from 

diverse  sources 

I  ncentivize  Traders 

Frequent  sampling  of 

information 


Mitigation  of  biases 

Shift  of  focus  from  estimating 

by  individuals  to  groups 
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Prediction  Markets 


•  A  place  where  people  can  buy  and  sell 
contracts  that  pay  the  owner  based  on 
some  future  event 


f  f  f  f  V  f  V  T  T 

Non -Ex  pert 

— n 

Good  Answer  Better  Answer! 
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Example 


•  Iowa  Electronic  Markets:  2008  Presidential 
Election 

-  Buy  and  sell  shares  of  candidate  votes 

-  Contract  pays  $0. 01  for  each  percentage 
point  of  Obama's  vote 

-  Contract  pays  $1.00  if  Obama  wins 


2008  Democratic  Nomination  Race 
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Hypothesis:  prediction  markets  are 
“efficient  markets" 


■  Market  prediction  summarizes  traders' 
beliefs  about  what  will  occur  in  future 


■  Truthful  revelation:  biased  trading  gets 
counter  traded  (e.g.,  political  trading  markets) 


■  I  nformation  discovery:  informed  trading  is 
rewarded 


I.K.a  il_  La-iJ  liI-  * 

— -  uMr 


■  Crowds  can  be  "wise" 


THE  WISDOM 
OF  CROWDS 


■  Diversity,  independence, 
decentralization,  aggregation 


JAMES 
SURO  WIECK1 
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Example:  Nielsen  ratings  for  Monday  night  football 
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%  of  115.9  million  television  households  in  the  United  States 


The  Case  for  Prediction  Markets 


•  "First  order  effects":  information 
aggregation  for  accurate  prediction 

-  Mechanism  for  bringing  together  information 

-  I  ncentives  for  searching  out  information 

-  I  ncentives  for  revealing  information 
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The  Case  for  Prediction  Markets 


•  "Second  order  effects":  organization 
culture  and  communication 

-  Signal  topics  of  interest 

-  Promote  interest  and  engagement  in  topics 

-  "Flatten"  hierarchy 

-  I  nvigorate  culture 
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Questions  PM's  can  answer: 

practical  guidelines 

•  "Contractible":  can  write  a  contract  about 
future  events  with  unambiguous  and 
verifiable  outcomes 

•  Outcome  categories  are  mutually  exclusive 
and  exhaustive 
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Types  of  Questions 


•  Bad:  "Sales  of  Xbox  consoles  will  exceed 
expectations  by  the  end  of  the  year. " 

•  Good:  "How  many  Xbox  consoles  will  be 
sold  between  November  1  and  December 
31,  2010?" 
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Using  Prediction  Markets:  Early 
Lessons 


•  Problem:  "I  don't  know  how  to  run  a 
prediction  market;  it  seems  hard" 

•  Remedy: 

-  Use  I  nkling  software: 
www.inklingmarkets.com 

-  Market  makers  are  key 
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Using  Prediction  Markets:  Early 
Lessons 


•  Problem:  "Liquidity":  not  enough  trades  and 
traders 

•  Remedies: 

-  Incentives: 

•  Prizes 

•  Recognition 

-  "Hard"  and  "Fun"  questions 

-  Newsletters 

-  Senior  management  involvement 
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Using  Prediction  Markets:  Early 
Lessons 

•  Problem:  "I  nsider  trading" 

-  Some  traders  have/will  have  inside 
information 

•  Remedies: 

Change  questions 

-  Change  trading  window 

-  Restrict  trading  access  to  "insiders" 
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Osama  Bin  Laden  to  be  captured/  neutralized  before  midnight  ET  on 
30  J  un  2011 


"  afmtjwrr 

Oct  08,  2010 -May  02,2011 
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Price 


Prediction  Markets:  Case  Study  -  SWCS 


Design  Components  to  consider 

Stocks 

Marketplace 

Traders 


IOC  FY  14 
FOC  FY  16 


Shallow  Water  Combat  Submersible 


I  mplement  prediction  markets  to  surface  potential 
program  risks,  and  generate  cost/schedule 
estimates  as  a  supplement  to  existing  estimation 
methodologies 
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Prediction  Markets:  Benefits  to  SWCS 


- \ 

Benefit  from  the  prediction  markets  event  forecasts. 

S _ J 


- \ 

I  ncreased  involvement  of  the  participants  in 

anticipating  events. 

- \ 

Identifying  informal  information  channels  in  their 

organization. 


- \ 

Continuous  review  of  factors  impacting  cost  and 

schedule 

< _ j 


Agicy| 
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Prediction  Market  Design 


Second  Order  Questions:  Traders  and  the  trading  process 


Who  has  information  about  the  program  (who  makes  money 
in  the  market?) 

Where  did  they  learn  this  information? 

What  is  your  motivation  for  trading?  (e.g.  to  win/to  solve  the 
problem/ to  validate  my  knowledge) 


Third  Order  Questions:  Behavior  outside  the  markets 


•  Did  prediction  market  participation  increase  team  knowledge, 
collaboration  and  information  sharing? 

•  Did  it  outperform  a  cost  model  in  terms  of  agility, 
transparency,  sensitivity  to  events? 
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Prediction  Market  Design  Principles 


Sufficiently  broad  following 


I  ncentives  based  on  constant  participation  and  ability 

to  predict  accurate  results 


Ample  historical  data 


Mix  of  hard  and  fun  questions 


Anonymity  of  participants,  information 
security/  confidentiality 
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